{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building Effective RNNs\n",
    "\n",
    "In this notebook, we discuss two topics that contribute towards building effective RNNs.\n",
    "- Increase The Representational Power of RNNs\n",
    "- Preventing Overfitting in RNNs\n",
    "\n",
    "\n",
    "\n",
    "## Increase The Representational Power of RNNs\n",
    "\n",
    "So far we have discussed how to train RNNs for very long input sequence. In addition to using weight initialization, proper activation, normalization, we introduced special RNN architectures such as GRU and LSTM.\n",
    "\n",
    "The next topic we discuss is how to create **powerful representations** using RNNs for solving complex problems. We discuss two techniques.\n",
    "\n",
    "- Deep RNNs\n",
    "- Bidirectional RNNs\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Deep RNNs\n",
    "\n",
    "All types of RNN architectures that we have discussed so far are based on a **single unidirectional hidden layer**. However, a single layer is quite limited in its capability to offer enough flexibility for modeling any function of interactions between the hidden state and the input sequence.\n",
    "\n",
    "Complex problems require us to build flexible models so that powerful representations can be created. In feedforward neural networks we did this by increasing the capacity of the model via stacking hidden layers. It enabled the network to learn more powerful representations. It was possible because each layer in a feedforward NN is connected to the next layer through nonlinearity. Thus the key technique is to stack computational units via nonlinearity.\n",
    "\n",
    "- How do we do this to increase capacity in RNNs? In other words, how do we stack the computational blocks in RNNs?\n",
    "\n",
    "\n",
    "One approach is to use multi-layer perceptron (MLP) inside the RNN cell. More specifically, we can include many hidden layers in the FC networks in a RNN cell (e.g., LSTM).\n",
    "\n",
    "Another approach is to stack multiple layers of RNN on top of each other. It will provide us a more flexible model because of the combination of multiple layers. Then, we could learn from input sequence in a hierarchical fashion. For example, lower layers may learn short-term dynamics and higher layers can focus on longer-term dynamics.\n",
    "\n",
    "Below we present the latter approach of stacking multiple hidden layers to build deep RNNs.\n",
    "\n",
    "\n",
    "<img src=\"https://cse.unl.edu/~hasan/Pics/RNN_DeepRNN_Architecture.png\" width=600, height=500>\n",
    "\n",
    "\n",
    "We describe the deep RNN architecture. Let's say that there are $L$ hidden layers. We denote the layer number using superscript $l$. The hidden state at layer $l = 1$ is computed as previously by using a minibatch input $\\pmb{X}_t$ at timestep $t$ and hidden state at the previous timestep $\\pmb{H}_{t-1}$: \n",
    "\n",
    "$\\pmb{H}_t^{(1)} = f_1(\\pmb{X}_t, \\pmb{H}_{t-1})$\n",
    "\n",
    "Then, the hidden states in the subsequent layers are computed:\n",
    "\n",
    "$\\pmb{H}_t^{(l)} = f_l(\\pmb{H}_t^{(l-1)}, \\pmb{H}_{t-1}^{(l)})$\n",
    "\n",
    "\n",
    "Finally, the output layer is only based on the hidden state of hidden layer $L$.\n",
    "\n",
    "$\\pmb{O}_t = g(\\pmb{H}_t^{(L)})$\n",
    "\n",
    "We can use any RNN architecture to implement this model, e.g., simple RNN, GRU, LSTM.\n",
    "\n",
    "## Deep RNNs: Many-to-One - TensorFlow Implementation\n",
    "\n",
    "Below we illustrate how to implement a 4-layer many-to-one RNN using a simple RNN model. We could also use GRU or LSTM in a similar fashion. A few things to note before designing a deep RNN model.\n",
    "\n",
    "- We do not need to specify the length of the input sequences (unlike in the feedforward neural networks such as FC MLP and CNN). Because a RNN can process any number of timesteps. Thus, we set the first input dimension to None.\n",
    "\n",
    "- We need to specify the feature length. Assume the length of the feature is 1 in this illustration.\n",
    "\n",
    "- By default, the SimpleRNN layer uses the tanh activation function.\n",
    "\n",
    "- Set \"return_sequences\" argument to True for all recurrent layers except the last RNN layer.\n",
    "\n",
    "- The final layer is a dense layer. It is applied only to the final timestep. Select a suitable activation for the last layer.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "simple_rnn (SimpleRNN)       (None, None, 20)          440       \n",
      "_________________________________________________________________\n",
      "simple_rnn_1 (SimpleRNN)     (None, None, 20)          820       \n",
      "_________________________________________________________________\n",
      "simple_rnn_2 (SimpleRNN)     (None, None, 20)          820       \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, None, 10)          210       \n",
      "=================================================================\n",
      "Total params: 2,290\n",
      "Trainable params: 2,290\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.SimpleRNN(units=20, return_sequences=True, input_shape=[None, 1]),\n",
    "    keras.layers.SimpleRNN(units=20, return_sequences=True),\n",
    "    keras.layers.SimpleRNN(units=20, return_sequences=True),\n",
    "    keras.layers.Dense(10)\n",
    "])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep RNNs: Many-to-Many - TensorFlow Implementation\n",
    "\n",
    "For creating a sequence-to-sequence model, we must apply the output Dense layer at every time step. For this we make the following changes.\n",
    "\n",
    "- Set \"return_sequences\" argument to True for all recurrent layers **including the last RNN layer**.\n",
    "\n",
    "\n",
    "- Apply the output dense layer to every timestep of its input sequence. Note that the dense layer is applied independently at each time step and that the model will output a sequence, not just a single vector.\n",
    "\n",
    "\n",
    "\n",
    "Alternatively, we could use Keras' TimeDistributed layer for this very purpose (code is commented out). \n",
    "\n",
    "\n",
    "### Keras' TimeDistributed Layer\n",
    "\n",
    "The TimeDistributed Layer wraps the desnse layer and applies it at every time step of its input sequence. It does this efficiently, by reshaping the inputs so that each timestep is treated as a separate instance. I.e., it reshapes the inputs from [batch size, timesteps, input dimensions] to [batch size × timesteps, input dimensions]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "simple_rnn_3 (SimpleRNN)     (None, None, 20)          440       \n",
      "_________________________________________________________________\n",
      "simple_rnn_4 (SimpleRNN)     (None, None, 20)          820       \n",
      "_________________________________________________________________\n",
      "simple_rnn_5 (SimpleRNN)     (None, None, 20)          820       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, None, 10)          210       \n",
      "=================================================================\n",
      "Total params: 2,290\n",
      "Trainable params: 2,290\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]),\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True),\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True),\n",
    "    keras.layers.Dense(10) # or use keras.layers.TimeDistributed(keras.layers.Dense(10))\n",
    "])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bidirectional RNNs\n",
    "\n",
    "\n",
    "So far, we have discussed unidirectional RNN models. Using these models, we can infer the next token based on the previous sequence of tokens. For example, based on the sequence \"The sky is\", an RNN model can learn the next word \"blue\".\n",
    "\n",
    "However, in some scenarios, the past sequence does not provide enough context to infer the next token. Consider the following two sentences.\n",
    "\n",
    "- The sky is (BLUE) on a sunny day\n",
    "\n",
    "- The sky is (DARK) at night.\n",
    "\n",
    "In these two examples, information on the color of the sky is encoded in the latter part of the sequences, after the color token. The uni-directictional RNN models perform poorly on this type of inference.\n",
    "\n",
    "One approach to overcome this limitation of the uni-directional RNNs is to build bidirectional RNNs (BRNNs), such that we can use information of both the past and future to make an inference.\n",
    "\n",
    "Following figure shows the architecture of a bidirectional RNN model with a single hidden layer. In this hidden layer, during forward propagation, information passes in two directions: from the start token to the end token, and then from the end toke to the start token. The BRNN model was first proposed in 1997.\n",
    "\n",
    "https://deeplearning.cs.cmu.edu/F20/document/readings/Bidirectional%20Recurrent%20Neural%20Networks.pdf\n",
    "\n",
    "There are various possible architectures of BRNN:\n",
    "https://www.cs.toronto.edu/~graves/ijcnn_2005.pdf\n",
    "\n",
    "<img src=\"https://cse.unl.edu/~hasan/Pics/RNN_Bidirectional.png\" width=600, height=500>\n",
    "\n",
    "## Computing the Hidden States in a BRNN\n",
    "\n",
    "We show the calculation of the hidden states in a single-layer BRNN. At each timestep there will two hidden states: forward $\\overrightarrow{H}_t$ and backward $\\overleftarrow{H}_t$\n",
    "\n",
    "\n",
    "$\\overrightarrow{H}_t = \\phi(\\pmb{X}_t \\pmb{W}_{xh}^{(f)} + \\overrightarrow{H}_{t-1} \\pmb{W}_{hh}^{(f)}) + \\vec{b}_h^{(f)}$  \n",
    "\n",
    "$\\overleftarrow{H}_t = \\phi(\\pmb{X}_t \\pmb{W}_{xh}^{(b)} + \\overleftarrow{H}_{t-1} \\pmb{W}_{hh}^{(b)}) + \\vec{b}_h^{(b)}$  \n",
    "\n",
    "Here the size of the minibatch input $\\pmb{X}_t$ is $m \\times d$ and the size of a hidden state is $m \\times h$. We have 4 weight matrices and two bias vectors for forward and backward processing.\n",
    "\n",
    "- Forward Weight Matrices and Bias Vector\n",
    "\n",
    "$\\pmb{W}_{xh}^{(f)}$: size $d \\times h$\n",
    "\n",
    "$\\pmb{W}_{hh}^{(f)}$: size $h \\times h$\n",
    "\n",
    "$\\vec{b}_h^{(f)}: size 1 \\times h$\n",
    "\n",
    "\n",
    "- Backward Weight Matrices and Bias Vector\n",
    "\n",
    "$\\pmb{W}_{xh}^{(b)}$: size $d \\times h$\n",
    "\n",
    "$\\pmb{W}_{hh}^{(b)}$: size $h \\times h$\n",
    "\n",
    "$\\vec{b}_h^{(b)}: size 1 \\times h$\n",
    "\n",
    "\n",
    "Note that we need to **concatenate** the forward hidden states $\\overrightarrow{H}_t $ and the backward hidden states $\\overleftarrow{H}_t$ for obtaining the hidden state $\\pmb{H}_t$. Due to concatenations, its size will be $m \\times 2h$. This hidden state is used as input to the output layer that computes the output as follows:\n",
    "\n",
    "$\\pmb{O}_t =\\pmb{H}_t \\pmb{W}_{ho} + \\vec{b}_o$\n",
    "\n",
    "Also note that in a BRNN, the forward and backward directions can have different number of hidden units.\n",
    "\n",
    "\n",
    "## When Should We Use BRNNs?\n",
    "\n",
    "\n",
    "BRNNs require information from both ends of the sequence to estimate the output. Thus, for making predictions, a BRNN will have to use information from both future and past observations to predict the current one. Let's say that we are building a language model that needs to predict the next word in a sequence based only on the previous words (e.g., speech recognition), then BRNNs won't be suitable. Because we don't have access to future data when we make inference. Thus, BRNNs will perform poorly in such scenarios.\n",
    "\n",
    "\n",
    "\n",
    "But for many natural language processing (NLP) tasks BRNNs turn out to be quite useful. For example, filling in missing words, annotating tokens (e.g., for named entity recognition). \n",
    "\n",
    "The NLP tasks at which BRNNs excel at is building embeddings is a self-supervised manner. To encode the words in a vocabulary, we can train a BRNN, which uses both the past and future context to learn powerful embeddings. These embeddings can then be used in downstream supervised tasks such as sentiment classification. This way BRNNs are used to create deep contextualized language models.\n",
    "\n",
    "https://arxiv.org/abs/1802.05365\n",
    "\n",
    "Due to their usefulness in many NLP tasks, BRNNs are known as the swiss army knife of NLP. But it's a very expensive piece of tool to afford. Let's see why.\n",
    "\n",
    "<img src=\"https://cse.unl.edu/~hasan/Pics/RNN_BRNN_SwissArmyKnife.png\" width=500, height=400>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Computational Cost of BRNNs\n",
    "\n",
    "BRNNs pass signals first in the forward direction, and then based on the outcome of the forward hidden states, a BRNN computes the backward hidden states. Finally, both the forward and backward hidden states are concatenated to create the complete hidden states for a single layer. Due to the dependency of the backward hidden states on the forward states, the loss gradients need to be propagated through a very long dependency chain. As a consequence, BRNNs are exceedingly slow to train. This computational cost blows up in case of deep BRNNs.\n",
    "\n",
    "This is the reason why the NLP community deflected from BRNNs irrespective of their representations prowess.\n",
    "\n",
    "\n",
    "## BRNN: TensorFlow Implemention\n",
    "\n",
    "We implement the BRNN model using Keras' Bidirectional layer. Then, we compare this model with a simple RNN model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"BRNN\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "bidirectional (Bidirectional (None, None, 40)          880       \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, None, 1)           41        \n",
      "=================================================================\n",
      "Total params: 921\n",
      "Trainable params: 921\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Model: \"RNN\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "simple_rnn_7 (SimpleRNN)     (None, None, 20)          440       \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, None, 1)           21        \n",
      "=================================================================\n",
      "Total params: 461\n",
      "Trainable params: 461\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model_brnn = keras.Sequential(name=\"BRNN\")\n",
    "model_brnn.add(keras.layers.Bidirectional(keras.layers.SimpleRNN(20, return_sequences=True), \n",
    "                                          input_shape=[None, 1]))\n",
    "model_brnn.add(keras.layers.Dense(1))\n",
    "\n",
    "\n",
    "model_brnn.summary()\n",
    "\n",
    "model_rnn = keras.Sequential(name=\"RNN\")\n",
    "model_rnn.add(keras.layers.SimpleRNN(20, return_sequences=True, \n",
    "                                          input_shape=[None, 1]))\n",
    "model_rnn.add(keras.layers.Dense(1))\n",
    "\n",
    "\n",
    "model_rnn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preventing Overfitting in RNNs\n",
    "\n",
    "We have discussed several techniques to increase the representational power of RNNs. The main idea is to build **large** models either by stacking many hidden layers or by passing information through both the forward and backward directions. This will essentially increase the model's capacity. But at the same time, the model will become prone to overfitting.\n",
    "\n",
    "Previously we used the **dropout regularization** in feedforward neural networks to prevent overfitting. In dropout regularization, the noisy correlations are broken by randomly assigning zeros to some input units. However, applying dropout in RNNs is not straight forward. \n",
    "\n",
    "In RNNs we have two types of input to produce the output.\n",
    "- Input data\n",
    "- Past hidden state\n",
    "\n",
    "Two weight matrices are associated with these two data types: \n",
    "- Weights of the input layer\n",
    "- Weights of the recurrent layers\n",
    "\n",
    "While applying dropout to the input weights is useful, applying dropout to the recurrent weights is not straight forward. If we drop some units in the recurrent layer randomly, then it will be detrimental for learning. Because the same units are applied to each input in a sequence. Randomly switching this units on and off will disrupt the loss propagation and will deteriorate the learning. Thus, stochastic on-off is not a useful dropout solution for RNNs.\n",
    "\n",
    "\n",
    "In 2015, Yarin Gal (as part of his PhD thesis) proposed a technique to apply dropout in the recurrent layers of RNNs. The main idea is that the same dropout mask (the same pattern of dropped units) should be applied to the recurrent units at every timestep, instead of a dropout mask that varies randomly from timestep to timestep. What’s more, in order to regularize the representations formed by the recurrent gates of layers such as GRU and LSTM, a temporally constant dropout mask should be applied to the inner recurrent activations of the layer (a recurrent dropout mask). \n",
    "\n",
    "This technique is effective because using the same dropout mask at every timestep allows the network to properly propagate its learning error through time. On the other hand, a temporally random dropout mask would disrupt this error signal and be harmful to the learning process.\n",
    "\n",
    "\n",
    "## Recurrent Dropout: TensorFlow Implementation\n",
    "\n",
    "Every recurrent layer in Keras has two dropout-related arguments: \n",
    "- dropout, a float specifying the dropout rate for input units of the layer\n",
    "- recurrent_dropout, specifying the dropout rate of the recurrent units\n",
    "\n",
    "Since networks being regularized with dropout always take longer to fully converge, we need to train the network for twice as many epochs."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
